Vision aid with three-dimensional image acquisition

ABSTRACT

The task of providing a device which makes it possible for the natural visual faculty of a partially vision-impaired person to be restored or supported as realistically as possible by means of a visual aid or visual prosthesis is achieved with the visual aid according to the invention in that, beyond the pure acquisition of a two-dimensional image, additional information is included in the image processing. For this purpose, a visual aid system is proposed which, during the reproduction of the processed image, visualizes additional information regarding the spatial position and/or specific attributes of objects in the acquired image which is associated with objects in the acquired image. The present invention thus affords the advantage that a three-dimensional image of the current environment of the user can be acquired owing to the fact that the image visualized to the user by the visual aid system according to the invention comprises additional information, for example, about the relative distance of the objects contained in the image and a depth-selective representation of the acquired image can thus be achieved.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.12/519,358 having a filing date of 15 Jun. 2009 and having a 371(c) dateof 23 Mar. 2010, which is a U.S. National Stage of InternationalApplication No. PCT/EP2007/011133 having an international filing date of18 Dec. 2007, which claims priority to German Patent Application No. 102006 060 045.2, filed on 19 Dec. 2006, each of the foregoingapplications are hereby incorporated by reference in their entirety.

BACKGROUND

The present invention relates to a vision aid with three-dimensionalimage and information acquisition. A vision aid in the sense of thepresent invention also encompasses other image acquisition systems, suchas e.g. residual-light amplifier or night vision apparatus for assistingthe human visual faculty. Such vision aids or visual prostheses, such ase.g. a retina implant, can also encompass functions for imageenlargement or for residual-light amplification.

Vision aids have already been developed in the form of implants for theretina of the human eye, which are provided for the treatment ofpatients, whose visual faculty has been partially or completely lostthrough defects in the retina. In the process, a microelectronic deviceis implanted in the region of the retina with a plurality oflight-sensitive pixel elements, on which an image projected onto theretina via the still-intact natural light path of the eye is received.In other visual prostheses, the image acquisition occurs through anexternal camera, in particular a video camera, which is preferablyaccommodated in spectacles. The image acquired through the pixelelements or the camera is converted into electrical signals anddelivered via stimulation electrodes by means of electrical stimulationimpulses to the ganglion cells of the retina and the optic nerve,thereby to restore or to improve the visual faculty of the patient. Theknown vision aids have the disadvantage, however, that they exclusivelyprocess the image acquired via the natural light path of the eye or fromthe external camera. The thereby resulting image is therefore merelytwo-dimensional and contains no three-dimensional information.

In order to restore or to assist the natural visual faculty by means ofa vision aid or a visual prosthesis as realistically as possible, it istherefore desirable to incorporate additional information in the imageprocessing beyond the pure acquisition of a two-dimensional image.

This task is solved by the inventive device having the featuresaccording to claim 1. Preferred embodiments of the present invention arecharacterized in the sub-claims.

SUMMARY

The present invention solves the above-mentioned task by a vision aidsystem with at least one camera for acquisition of an image having aplurality of image data, electronic means for processing the acquiredimage data, and a stimulation device implantable in the body at thevisual organ for reproduction of the processed image via correspondingstimulation of the visual organ, wherein, in the reproduction of theprocessed image, additional information and/or particular attributesregarding the spatial position of objects in the acquired image arevisualized, these being associated with objects in the acquired image.

With a vision aid according to the present invention, objects in theimmediate vicinity of the user can in particular be specially visualisedor accentuated, while more distant objects can optionally be masked out.The accentuation of particular objects can also occur through the use ofdifferent grey-scales or varying colours. In this way, objects locatedin the near vicinity may, for example, be represented in another colouror grey-scale than more distantly lying objects. The present inventionthus offers the advantage that a three-dimensional copy of the currentenvironment of the user can be acquired, in which the image visualizedby the user from the vision aid system according to the inventioncomprises additional information, for example, about the relativedistance of the objects contained in the image and may thereby achieve adepth-selective representation of the acquired image.

The advantage of the depth-selective representation resides in that theuser of the vision aid system can receive a visually well-structured andeasily interpreted visual impression of the near or more distantvicinity despite a limited pixel resolution, colour graduation, greyshading or limited member of picture elements (pixels) of the vision aidsystem. The vicinity of the user can then, for example, be visualisedvia an outline representation, which is able to be realised with arelatively low number of simultaneously activated picture elements.

According to the present invention, the entire system of the vision aidcomprises spectacles and optionally an additional unit carried on thebody, e.g. a pocket computer, which can be carried at any desiredposition of the body and can communicate and exchange image data withthe spectacles via a cable connection or alternatively via wirelessradio connection. In a special application of the present invention toan above-described retina implant system, a device implanted in the bodycan additionally be provided, which receives the image data acquired bythe external camera.

In a preferred embodiment of the present invention, a three-dimensionalmap for preparation of the depicted image section is employed. Athree-dimensional map of the vicinity of a particular image section isable to be employed in the form that only the segments of the image aredepicted that are present in the near or middle distance of the user ofthe vision aid, because these may be of greater interest for the userthan segments of the image or objects, which are located at a greaterdistance. Alternatively, in a special operating mode of the vision aidaccording to the invention, only more distant objects may be depicted,in the event the user of the vision aid preferably wants to visualisethe background of the acquired image or the more distant objects.

The different operating modes of the vision aid can either be rigidlyimplemented in the image processing of the system or be availableoptionally selectable for the user. The selection of the operating modescan be effected via activation elements of the spectacles, via theoptionally provided pocket-computer, or via a remote control, which maybe worn at any desired position on the body.

The display area of the system is able to be set through correspondinglyselected mathematical intersection surfaces or surface functions in athree-dimensional model of the pocket computer. As intersectionsurfaces, sections of planar surfaces or a cylindrical or a sphericallysurface or other desirably formed intersection surfaces may, forexample, be selected. According to the operating mode of the implantsystem, only the intersection surfaces with the acquired objects arethereby depicted in the three-dimensional model of the pocket computer(or the spectacles).

Through a setting device of the implant system, the user of the visionaid can select such an operating mode, for example, a scan mode, withwhich only those objects are depicted that are located in a particularacquisition area of the system. Further, the user of the vision aid canvary the optical acquisition area of the system by means of the settingdevice. In a particular operating mode of the system according to theinvention, the optical acquisition area of the system can be repeatedlyexpanded and then contracted again, so that the user of the vision aidcontinuously receives spatial section images of the environmentpresented.

According to a further preferred embodiment, the vision aid according tothe invention can, by means of the electronic calculating capacity ofthe pocket computer with the acquired image data, generate athree-dimensional map and visualize this to the user. With athree-dimensional map, an image section can also be presented to theuser of the vision aid, for example, in a birds-eye perspective, inwhich an outline of the current environment and the objects locatedtherein is depicted, which can be especially helpful for navigation inan unclear environment.

These operating modes may not only be employed with vision aids forusers having extremely reduced visual capacity, but also invisualization systems for orientation systems for users with normalvisual capacity. The present invention can also be applied in visionaids in which a video camera integrated in the spectacles acquires theimage and is projected onto the retina of a vision impaired user via adisplay also integrated in the spectacles. In the process, even coarsepixel presentations may achieve an improvement of the visual impression.

In a further preferred embodiment of the vision aid according to theinvention, a three-dimensional map is generated by use of only onecamera. In the process, the camera which is integrated in the spectaclesof the user for acquisition of the image information is preferably used,in order to carry out a stereoscopic image processing. In thisconnection, images from different spatial positions of the camera can beused, which have been recorded after one another with slight timeoffset. For this purpose, the relative spatial positions and therelative angular orientations of the camera must be taken into account,which will be explained in the following section.

The spectacles, in which the camera is integrated, may for example beequipped with three-dimensional acceleration sensors, which respectivelyascertain the acceleration [{umlaut over (x)} ÿ {umlaut over (z)}] ofthe spectacles and thereby the acceleration of the camera in the threespatial directions [x, y, z]. Alternatively, the spectacles in which thecamera is integrated may be equipped with a six-dimensional accelerationsensor, which respectively ascertains the acceleration of the spectaclesand thereby the acceleration of the camera in the three spatialdirections [x, y, z] as well as the angular acceleration [{umlaut over(x)} ÿ {umlaut over (z)} {umlaut over (φ)}_(x) {umlaut over (φ)}_(y){umlaut over (φ)}_(z)] of the reference system of the spectacles aboutthe three spatial axes. The dimensionality of a sensor consequentlyspecifies the number of the measured variables that a sensor mayascertain. A six-dimensional acceleration sensor ascertains, forexample, three translational accelerations and three angularaccelerations, which have an orthogonal reference to one another andthereby ascertains a total of 6 orthogonal acceleration measurements.

In order to determine the current camera position relative to a previouscamera position, an inertial navigation method may be used. For thispurpose, one, two or more acceleration sensors (one-dimensional,two-dimensional, three-dimensional or multi-dimensional, e.g.six-dimensional acceleration sensors can be placed at kinematicallywell-conditioned positions in the spectacles. Kinematicallywell-conditioned positions are, for example, such positions that aresufficiently well spaced from one another within the spectacles.

Through double integration of the above-mentioned six orthogonalacceleration measurements, the relative position [Δx Δy Δz Δφ_(x) Δφ_(y)Δφ_(z)] of the spectacles or the camera position can be determinedrelative to a previous camera position. The integration of theacceleration measurements and the calculation of the position andorientation of the spectacles occurs advantageously through electronicintegrators in a calculator unit integrated in the vision aid system,which is accommodated either in the spectacles or outside of thespectacles, e.g. in a unit (“pocket computer”) carried on the body ofthe user, or can be distributed onto the spectacles and the unit carriedon the body. In so doing, the unit carried on the body can communicatewith the spectacles either per cable connection and/or via wirelessconnection (e.g. by means of Bluetooth, WLAN or other communicationtechnology).

The time integration increment in the integration of the accelerationmeasurements should be sufficiently smaller than the time span withinwhich the acceleration of the spectacles changes substantially, in orderto be able to measure sufficiently precise changes in position of thespectacles or camera over a limited time period, e.g. 10 seconds, withwhich then at least two or more camera images can be acquired fromdifferent positions of the camera.

By means of the above-mentioned double integration, a six-dimensionalrelative position vector is generated from the six orthogonalacceleration measurements of a 6D-acceleration vector, which specifiesthe difference of a previous position of the camera to a subsequentcamera position. This six-dimensional relative position vector containsthree location differential coordinates (Δx, Δy, Δz), which specify thedeviation of the location coordinates (x, y, z) of the previous cameraposition to the subsequent camera position, and three angulardifferential coordinates (Δφx, Δφy, Δφz), which specify the deviation ofthe angular position (φx, φy, φz) of the previous camera position to thesubsequent camera position. The integration can be effected in discretetime increments or can be carried out continuously.

In this way, a sequence of camera images at respectively differentcamera positions can be acquired, wherein a sequence consists of atleast two camera images. A relative position in reference to theprevious or the subsequent camera image is assigned to each camera imageof this sequence, for example, by means of the above-mentioned locationdifferential coordinates and angular differential coordinates. Cameraimages can also be recorded in fixed time intervals with correlatedposition, wherein older images can optionally be increasingly moreweakly weighted in the influence on the current spatial calculationuntil they ultimately have no further influence. In this way, the effectthat the position determination by means of the inertial navigation overgreater time spans can include increasing inaccuracies in the form of aposition drift is counteracted.

In a further preferred embodiment of the vision aid according to thepresent invention, an assignment of camera images and camera positionsto different measurement time points can be undertaken. Thus, a secondcamera, which would otherwise be needed for taking stereo images, can beadvantageously saved without dispensing with three-dimensional images.Further, more than two images from different positions and orientationscan be processed, in order to thereby generate a more completethree-dimensional map of the environment of the user of the vision aid,which may take place e.g. by means of the known “batch compensationmethod”. A prerequisite for this method are small head movements of theuser in the order of magnitude of the eye separation of two human eyes.The camera may then respectively acquire one image from differentpositions and thereby deliver the two images to the calculator unit inshort time succession, which a human with normal visual capacity wouldacquire with both eyes. The calculator unit can thereupon separatelyprepare the two camera images and update and expand thethree-dimensional model in the pocket computer.

Furthermore, at the beginning of the above mentioned integration, ameasurement of the speed-offsets can be carried out in order to improvethe measurement accuracy and thus the allocation of the acquired cameraimages to the calculated local- and angular-positions of the camera.This can occur in different possibilities:

The user of the vision aid may, through corresponding input means, setthe speed of the spectacles to zero, e.g. upon keeping the spectaclesstill for a short time period. This can, for example, also occur throughbrief placement of the spectacles onto a motionless support. Withsufficiently high accuracy of the acceleration sensors and theelectronic integrators, it can also suffice to reference the spectacleswith the speed-offset zero during the charging process, where they arewith high probability at rest, and to start the integration from thistime point and to employ this speed-offset for the entire daily use.

The current absolute speed can also be determined with the aid of othernavigation resources, for example, by means of GPS or similar radiotriangulation methods. Further, the current speed of the spectacles orthe user of the vision aid can be ascertained via the change of theimage information in combination with the acceleration measurement dataof the spectacles. In addition to the above-mentioned speeddetermination, the possibility also exists to determine the currentposition of the spectacles of the user by means of GPS or similar radiotriangulation methods. The orientation of the spectacles in space, i.e.the angular positions of the spectacles, can be ascertained throughevaluation of the previous angular position values or e.g. with acompass (e.g. a Hall probe) or through similar orientation measurementmethods. The angle about the nodding axis of the spectacles (inclinationangle), which is not ascertainable with a compass, may for example beascertained via a gravitation-based inclination sensor.

A combination of the above-mentioned methods can also be employed, inorder to minimize the measurement error of the sensors integrated in thespectacles, particularly in order to specifically treat exceptions thatcan arise with the different methods. This applies in particular inevent that parts of the environment of the user move quickly or the useris located in a fast-moving inertial system, like e.g. a train, car, oraircraft.

According to a further preferred embodiment of the vision aid accordingto the invention, the preparation of stereoscopic or three-dimensionalinformation with respect to the spatial environment in the visual fieldof the user occurs through the use of a second camera in the spectacles.In the process, the second camera is preferably also integrated in thespectacles and spaced as far as possible from the first camera in thespectacles. Because of the known, fixed separation of the two camerasfrom one another, stereoscopic depth information about segments of thecurrent image section is able to be calculated from the two slightlydifferent images, which can then be converted by the electroniccalculator unit to a three-dimensional map of the environment of theuser and conveyed to the user via stimulation means.

In a still further preferred embodiment of the vision aid according tothe invention, distance sensors are alternatively or additionallyemployed, such as e.g. ultrasound sensors, rotating laser scanners inthe visible or invisible wavelength range, e.g. in the ultraviolet orinfrared light range. With the aid of such distance sensors, theenvironment of the implant-wearer can be better captured and therebyobstacles also better identified. The information acquired from thedistance sensors can, for example, be further processed withtriangulation methods and assigned to corresponding sections of theacquired camera image or the camera image sequences. In this respect,multiple sensors may e.g. be arranged in the form of a one-dimensionalor multi-dimensional array. In this preferred embodiment of the presentinvention, the spectacles comprise one- or multi-dimensional arrays ofsensors or actuators, such as e.g. ultrasound sensors, infrared sensors,photo-sensors, lasers or other sensors or actuators or combinationsthereof

In a further preferred embodiment of the vision aid according to theinvention, a stripe projection or structured illumination is carriedout. In the process, a homogenous or a structured (e.g. “gray coded”)stripe-pattern is projected onto the environment of the user, preferablywith infrared light, e.g. with laser light. The generation of a stripepattern occurs, for example, through a projection template, which ispositioned in front of a light beam. In this connection, interferenceeffects can alternatively e.g. with use of a laser beam beadvantageously used in order to generate a corresponding pattern in theenvironment of the user. Alternatively, for example, a light beam can bediverted with the aid of a micro-mirror system and paralleltime-modulated to capture the current visual field of the user.

The stripe pattern is invisible with the use of infrared light. Thecamera in the spectacles of the vision aid is, however, sensitive toinfrared light and can detect the stripe pattern. Distortions of thisstripe pattern (e.g. deflections, displacements) can be transformed backthrough corresponding triangulation methods into dimensional depthinformation, through which a three-dimensional map of the current imagesection is, in turn, able to be calculated.

In this connection, the stripe pattern may also be periodicallyactivated in defined time sections which lie between the time points atwhich the original video image is recorded (e.g. 10 ms stripe pattern,490 ms no stripe pattern). The actual image of the environment is herebyalso simultaneously available, which can, for example, be applied for aso-called “texture mapping” of the current environment onto thecurrently depicted object. Instead of a “texture mapping”, the currentlynot-displayed objects can also be masked out with the three-dimensionalmap, so that only the image sections of the currently depicted objectsare visible, which is accomplishable with relatively low computingeffort. In this way, the environment of the user of the vision aid canbe depicted more clearly.

In a further preferred embodiment of the vision aid according to theinvention, the spectacles are equipped with a camera and a gridprojector, such as e.g. a laser projector with deflection unit, with aprojection mask having a light source or other projection means. In thisembodiment, the sampling of the visual field detected by the vision aidsystem occurs by means of a laser beam, which is positioned with acorresponding deflection unit in the beam path of the laser, such ase.g. a piezo-technology driven micro-mirror system or rotating mirrorsystem. The camera image can thereby be used for position determinationof the picture element, from which the position of the reflecting objectsegment is then, e.g. through triangulation, again able to beascertained. Alternatively, a fast XY-photo-detector can also beemployed, which can ascertain the current position of the pictureelement in the environment of the user and thereby make possible asampling of the entire user environment in the range of seconds.

In a further preferred embodiment of the vision aid according to theinvention, a time duration measurement of the reflected laser light isundertaken with the aid of a fast photo-detector. The time durationbetween the moment of transmission of the respective laser light impulseand the moment of reception of the reflected laser light impulse in thephoto-detector is hereby ascertained and, taking account of the speed oflight, the distance of the related picture element is calculatedthere-from. Afterwards, a three-dimensional map of the related imagesection is able to be constructed through evaluation of the measurementsfor the image raster.

According to a further preferred embodiment of the present invention,the three-dimensional map generated by the pocket computer can also befiled in an external three-dimensional data storage, wherein particularobjects, critical terrain shapes or important items in the image areaacquired by the vision aid system can be specially characterized oraccentuated. That way, the following particular objects of an everydayenvironment can, for example, be particularly characterized in thethree-dimensional map of the vision aid system: staircase steps,light-posts, traffic lights, zebra crossings, curb-stone edges, groundunevenness, motor vehicles, bicycles or other vehicles of differenttypes and shapes, people, faces, body parts, door frames, window frames,tables, crockery or other important objects, which require theparticular attention of the observer or the user of the vision aidsystem.

The particular objects are preferably automatically detected bycomparison with an image library stored in the pocket computer or in theexternal data storage. This image library may be filled with any desiredimage samples that could be of importance to the user of the vision aidsystem, such as e.g. pictures of a traffic light or of other importantobjects from various viewing directions. Through permanent comparison ofthe video images acquired or recorded by the vision aid system with theimage samples saved in the image library, these particular objects inthe video image can be automatically recognized and thereaftercharacterized in the three-dimensional map.

The acquisition of image components in the three-dimensional mapgenerated by the pocket computer can, for example, take place throughthe known Gabor filtering or other methods of image segmentation. Forthis purpose, for example, each picture element vector Pi=(xi, yi, zi)of the three-dimensional map can be provided with a characteristicvariable Ei, from which an expanded picture element vector Pi=(xi, yi,zi, Ei) with (i=1, . . . , N) results. This characteristic variable Eimay represent either simple attributes in binary manner, such as e.g.important or unimportant, or also an grading in different classes, suchas e.g. not interesting, interesting, very interesting, helpful ordangerous. Objects with one of these attributes or characteristics canthen be visualized in the image section depicted by the vision aidsystem by means of special effects, e.g. by flashing, a representationwith greater brightness, by special coloration or other markings

According to a further preferred embodiment of the vision aid systemaccording to the invention, the distance measurement of picture elementsor objects in the environment of the user occurs with the aid of a laserdistance sensor. Such a laser distance sensor typically comprises alaser beam rotating around the z-spatial axis, which can sample theenvironment line-by-line with each revolution, wherein, depending uponthe mounting position, a sampling of up to a complete angularcircumference of 360° is possible. The laser beam source is positionedon a rotating platform, which for its part may acquire the currentrotational angle z of the platform (0° to 360°.

The laser source of a laser distance sensor transmits laser lightimpulses in time intervals sufficiently far spaced from one another,which impulses are reflected from objects in the scanned environment andwhose return at the location of their transmission is time-measured bymeans of a photo-detector. The capture of the emitted and reflectedlight impulse is, for example, carried out with the aid of a photodiodeand the time duration of the light impulse ascertained with sufficientlyfast electronics. From the measured time interval (duration t) betweenthe emission of the laser impulse and the return of the laser impulseand the speed of light (c=3 108 m/s), the distance s of the objectstruck by the laser pulse is able to be calculated from the productaccording to the formula s=½ v t. An advantage of the distancemeasurements by means of a laser distance sensor resides in that, a verysmall picture element can be projected by the laser beam onto theenvironment, such that a detailed scanning and resolution of the objectsurface is able to be obtained.

In order to obtain a complete image of the environment and to acquirethe entire image surface, it is necessary to sample more than one imageline. For this purpose, the laser beam of the laser distance sensor mustbe additionally deflected in the vertical direction, in order to samplethe image columns in a desired angular section. This can, for example,take place with a mirror pivotable about the x-axis, which is similarlyequipped with an angle measuring device for the angle x about thex-spatial axis. With the angles x, z and the distance s, the positionPS=(xS, yS, zS) of the sighted picture element is able to be calculateddirectly via triangulation and to be adopted as spatial point in thethree-dimensional map generated by the vision aid system.

As laser source, a so-called VCSEL (Vertical Cavity Surface EmittingLaser) is, for example, able to be used. Such a component exhibits asmall edge length of about 300 μm and is therefore well able to beintegrated in the spectacles of the vision aid system. As verticaldeflection unit, a commercially available micro-mirror can, for example,be used, such as that already employed e.g. in video-projectors. Therotation of the micro-mirror about the z-spatial axis may, for example,occur with a rotating, mirrored prism, which is driven by a micro-motor.Dimensions for the entire assembly can thus be achieved below 10 mm edgelength, which can be accommodated in the spectacles of the vision aidsystem. If a VCSEL is used with infrared laser light outside of thevisible spectral range, the sampling of the environment can thus occurwithout persons in the vicinity of the user being disturbed or takingnotice.

As described above, the three-dimensional information acquisition andfurther processing of the acquired image data of the vision aid systemcan be supported by an externally generated three-dimensional map of theacquired image data. The three-dimensional map is preferably generatedby means of the electronic components of the pocket computer and storedon the pocket computer or on an external three-dimensional data storage,such as e.g. an electronic server, in wireless connection with thevision aid system (e.g. via WLAN or Bluetooth).

The three-dimensional map can also be made available as electronic mapmaterial via external sources, for example, through a map for navigationsystems which are transferred to the server or the pocket computer via aCD or other storage medium or per Internet connection. With suchelectronic map material, the environment is, as a general rule, picturedin the form of grid networks onto which an image of the world surfacerecorded by video can optionally be wrapped or projected.

With the three-dimensional map, it can also concern three-dimensionalinformation specially produced or prepared for a retina implant systemwhich contains a detailed profile of the near vicinity. The adequateminimal resolution of such a depiction preferably lies in the order ofmagnitude from a few millimetres to centimetres. The three-dimensionalmap can, for example, contain N individual grid points Pi=(xi, yi, zi)with (i=1, . . . , N), which correspond with particular pixels orsurface points of the related environment. These grid points can beconnected with neighbouring points by lines in order to thereby span agrid network.

An electronically recorded image, for example, in the form of aphotograph of the environment, can be wrapped or projected onto thesurfaces spanned in this grid network in the three-dimensional map. Thisoccurs advantageously with the known method of so-called “texturemapping”, with which a three-dimensional picture of the environment isgenerated. With the position PB=(xB, yB, zB) and orientation B=(Bx, By,Bz) of the observer or of the spectacles of the user of the vision aidsystem, the currently visible field of vision of the user can becalculated with the aid of known standard projection, scaling, androtation methods from matrix-algebra. Such methods are already knownunder the names “ray-tracing” (beam retracing), “vector graphics” or“texture mapping” (image projection on three-dimensional grid networkbodies).

The position PB and orientation B of the observer or of the spectaclesis ascertained with the aid of the pocket computer in order to representthe currently acquired image section. This occurs with the inertialnavigation method described above or e.g. via GPS navigation. A precise,locally referenced GPS-like navigation method can also be used, whichcan also determine the position PB of the observer sufficientlyaccurately in the range of millimetres or centimetres via radiotriangulation through a measurement of time duration differences bymeans of a number M of reference transmissions. The exact absoluteposition PRj=(XRj, YRj, ZRj) with (j=1, . . . , M) the referencetransmitter is thereby preferably stored in the pocket computer.

The spatial direction or the orientation B of the spectacles of thevision aid system according to the invention can also be determined viaa gravitation based inclination sensor or other method (e.g. radiotriangulation). The acquired video image of the camera in the spectaclesmay optionally be combined with the three-dimensional grid network modeland be used e.g. for the “texture mapping”. In this way, the potentiallydistracting background of the video image can be masked out in order todepict only the objects located in the vicinity of the user. Alldistracting background parts of the image may thus be depicted e.g.black, i.e. no electrical stimulation takes place at these positions inthe implant. In doing so, the objects lying in the foreground of thethree-dimensional grid network model deliver, with their profile, theborderlines to the sections of the video image which are to be depicted.

The generation of the three-dimensional map for the environment of theuser can also be carried out by means of the electronic components,which are not integrated in the spectacles of the vision aid system. Thevideo images acquired by the spectacles and the respectivelycorresponding position PB and orientation B of the spectacles can thenbe conveyed to the pocket computer or to a three-dimensional server orper radio or a cable connection, which can generate or update thethree-dimensional map of the environment with these data. In addition,the so-called batch compensation method or other methods can, forexample, be employed.

For the above-mentioned inertial navigation, acceleration sensors andelectronic integrators are in principle applied, wherein themeasurements of the acceleration sensors are integrated by means of theelectronic integrators. An acceleration sensor normally consists of anaccelerated reference mass M, which is elastically mounted in a carrier.The position of the mass relative to the carrier can be capacitivelydetermined with comb-like measurement structures. For this purpose, thecomb-like measurement structures are fixed to the carrier as well as tothe reference mass. In dependence upon the distance s, around which theintermeshing comb-like measurement structures displace against eachother, a change in the electrical capacity of the arrangement resultsand thereby a corresponding measure for the mechanically acceleration a.With known spring strength D of the mounting of the reference mass, thespring force results from the formula F=D·s and, by equating the knownformula for the acceleration force F=m·a, the measure for theacceleration a of the reference mass results with a=(D/m)·s. Suchsilicon acceleration sensors manufactured in semi-conductor processeshaving small dimensions are already available in three-dimensionalformats, such as e.g. the component “ADXL 330” of the company AnalogDevices.

With the method of inertial navigation, it is advantageous to mount theacceleration sensor in a gyroscope rotatable about all three spatialaxes in order to obtain a gyroscopic stabilisation, i.e. a selectedfixed spatial orientation of the arrangement. The number of the requiredacceleration sensors can thus be reduced to a single three-dimensionalxyz-sensor.

An electronic integrator can be constructed either analogue with the aidof precise operation amplifying circuits or can be realised with theadoption of a digital processor having diverse mathematicallyintegration methods, which preferably adapt their precision to the inputsignal. Some known integration methods are, for example, the Eulermethod, the Runge-Kutta method, the Bulirsch-Stoer method and theAdams-Gear method. For the realization of an electronic integrator,methods can also be applied that are base upon modifications orcombinations of the above-mentioned methods. A numerical integrationmethod is, as a general rule, more precise than electrical integrationmethods and is therefore preferably employed with the vision aid systemaccording to the present invention.

In the following, the present invention is more closely explained basedon the preferred embodiments with reference to the accompanyingdrawings. They show:

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 a schematic representation of the vision aid system according toa preferred embodiment of the present invention with a cable connectedcoupling between the spectacles and the pocket computer-version;

FIG. 2 a schematic representation of the vision aid system according toa further preferred embodiment of the present invention with a wirelesscoupling between the spectacles and the pocket computer;

FIG. 3 a schematic representation of the vision aid system according toa further preferred embodiment of the present invention withthree-dimensional or n-dimensional acceleration sensors, where n=1, 2,3, 4, 5, 6, . . . ;

FIG. 4 a schematic representation of the vision aid system according toa further preferred embodiment of the present invention with asix-dimensional acceleration sensor;

FIG. 5 a schematic representation of the calculation carried out in thevision aid system according to the invention for position determinationof the vision aid system according to a preferred embodiment of thepresent invention;

FIG. 6 a schematic representation of a movement trace with a number ofdifferent successive positions of the vision aid system according to thepresent invention;

FIG. 7 a schematic representation of the vision aid system according toa further preferred embodiment of the present invention with two camerasfor acquisition of a stereoscopic image;

FIG. 8 a schematic representation of the vision aid system according toa further preferred embodiment of the present invention with sensorarrays; and

FIG. 9 a schematic representation of the vision aid system according toa further preferred embodiment of the present invention with a gridprojector.

DETAILED DESCRIPTION

In FIG. 1 a schematic representation of the vision aid system 1according to a preferred embodiment of the present invention is shown.In this embodiment, the vision aid system 1 according to the inventioncomprises spectacles 2 which, in their shape, resemble normal spectaclesand can also be worn by the user of the vision aid system 1 like normalspectacles. An electronic camera 7 which acquires video images in thefield of vision of the vision aid system and transforms them intoelectrical image information with picture elements (pixels) isintegrated in the spectacles 2. In the embodiment illustrated in FIG. 1,the entire system 1 of the vision aid according to the inventioncomprises spectacles 2 and a pocket computer 3 which can, for example,be carried as a separate unit on the body of the user. The spectacles 2and the pocket computer 3 are connected with one another via a dataconductor 4, which can also optionally encompass a power conductor forconveying power from the pocket computer 3 to the spectacles 2.

The vision aid system according to the invention comprises electronicmeans for electronic processing of the acquired image data. Theseelectronic means are preferably accommodated in a calculating unitintegrated in the vision aid system and/or in the separate unit 3, whichcan exchange image data and/or control data with the camera via the dataconductor 4. In the embodiment illustrated in FIG. 1, electroniccomponents are contained in the separate element or the pocket computer3, which can electronically process the image data acquired by thecamera 7 and transmitted via the data conductor 4. The image dataprocessed by the electronic components in the pocket computer 3 can bemade visible on a display (not shown) and be viewed there by the user.

In an application of the vision aid according to the invention forassisting a vision-impaired user, the image data processed by theelectronic components in the pocket computer 3 may be transmitted backto the spectacles 2 via the data conductor 4 and relayed to themicroelectronic stimulation device 6 implanted in the eye 5 of the user.The transfer of the image data from the spectacles 2 to themicroelectronic stimulation device 6 takes place, for example,wirelessly via an inductive and/or optoelectronic interface. Themicroelectronic stimulation device 6 transforms the received image datainto electrical stimulation impulses and delivers these to the ganglioncells of the retina and the optic nerve, whereby a corresponding imageis visualized to the user.

In addition to the realistic reproduction of the image acquired by thecamera 7, particular objects in the acquired image can, throughadditional information, be specifically accentuated and visualized tothe user with the vision aid system according to the invention. Theaccentuation of particular objects can, for example, take place throughuse of different grey-scales or through use of particular colours forthe visualization of the related objects. Furthermore, particularlyobjects in the direct vicinity of the user can be specially accentuatedthrough the visualization of information on the distance to the relatedobjects. In this way, for example, objects located in the near vicinitycan be depicted in another colour or grey-scale than more distantlylying objects. Additionally or alternatively, more distant objects canbe masked out of the acquired image. Furthermore, the environment of theuser can, for example, be visualized via an outline depiction.

In an application of the vision aid according to the invention inconnection with an above-described retina implant system for assistanceof a vision-impaired user, a microelectronic stimulation device 6, whichreceives the image data acquired by the external camera 7, can beimplanted in the eye 5 of the user. The micro-electronic stimulationdevice 6 is implanted in the region of the retina and comprises aplurality of stimulation electrodes which deliver electrical stimulationimpulses to the retina corresponding to the received image data. In thisway, the image acquired by the camera 7 is converted into electricalsignals and delivered to the ganglion cells of the retina and the opticnerve via stimulation electrodes of the stimulation device 6 by means ofelectrical stimulation impulses in order to restore or to improve thevisual faculty of the user.

The spectacles 2 and the separate unit or the pocket computer 3 of thevision aid system 1 are connected with one another via a cabled dataconductor and/or power conductor 4 and can transfer data in bothdirections over this communication path. In one direction, the imagedata acquired by the electronic camera 7 in the spectacles can betransmitted to the pocket computer 3 and, in the other direction, thepocket computer 3 can, for example, transmit control data or stimulationdata to the camera 7. These control data sent to the camera 7 from theseparate component 3 serve, for example, for setting the sharpness, thedirection, the focus or the zoom of the camera 7, the selection or theenlargement of a particular image section that the camera 7 shouldacquire.

FIG. 2 shows a schematic representation of the vision aid systemaccording to a further preferred embodiment of the present inventionhaving a wireless communication path. The configuration of theembodiment of the vision aid system according to the inventionillustrated in FIG. 2 essentially corresponds to the configuration ofthe embodiment depicted in FIG. 1, so that reference can be made to thedescription of FIG. 1. As distinguished from the previously describedembodiment, a wireless data conductor 8 is provided in the embodimentdepicted in FIG. 1 [sic] instead of the cable conductor 4. Such awireless data conductor 8 may, for example, be accomplished as abi-directional radio connection between the spectacles 2 and theseparate unit or the pocket computer 3 of the vision aid system 1. Viathe bi-directional wireless data conductor 8, the spectacles 2 and theseparate unit or the pocket computer 3 of the vision aid system 1 cancommunicate with one another and exchange image data or control data.

FIG. 3 shows a schematic representation of the vision aid systemaccording to a further preferred embodiment of the present inventionwith three-dimensional acceleration sensors 9. There, both the camera 7as well as three acceleration sensors 9 are arranged on the spectacles 2and, in this way, are coupled with one another in the movements. Theacceleration sensors 9 measure the translational acceleration [{umlautover (x)} ÿ {umlaut over (z)}] of the camera 7 in the three spatialdirections [x, y, z]. Additionally, the angular acceleration [{umlautover (x)} ÿ {umlaut over (z)} {umlaut over (φ)}_(x) {umlaut over(φ)}_(y) {umlaut over (φ)}_(z)] of the camera 7 about the three spatialaxes can be ascertained from the translational acceleration values ofthe sensors 9.

FIG. 4 shows a schematic representation of the vision aid systemaccording to a further preferred embodiment of the present inventionwith a six-dimensional acceleration sensor. The six-dimensionalacceleration sensor 10 is, like the camera 7, arranged on the spectacles2 and thereby performs the same movements and accelerations as thecamera 7. This six-dimensional acceleration sensor 10 is able to measureboth the translational acceleration [{umlaut over (x)} ÿ {umlaut over(z)}] of the camera 7 as well as the angular acceleration [{umlaut over(x)} ÿ {umlaut over (z)}{umlaut over (φ)}_(x) {umlaut over (φ)}_(y){umlaut over (φ)}_(z)] of the camera about the three spatial axes.

With the measurements delivered by the acceleration sensor 9 and theangular acceleration sensor 10, a change of the spatial orientation ofthe camera 7 relative to a previous spatial orientation of the camera 7can be determined. To this end, the spatial position or the orientationof the camera 7 relative to a previous position or orientation of thecamera 7 is determined by means of the electronic means of the visionaid system 1 by use of an inertial navigation method.

FIG. 5 shows a schematic representation of the calculation carried outin the vision aid system according to the invention for the positiondetermination of the vision aid system according to a preferredembodiment of the present invention. The electronic means of the visionaid system 1 according to the invention comprises electronic integrators11 and 12, which are able to numerically execute a mathematicallyintegration. As illustrated in FIG. 5, a six-dimensional accelerationvector is able to be generated from the orthogonal accelerationmeasurements measured by the acceleration sensors, which vector containsthe three translational acceleration values and the three angularacceleration values. The indices x, y, z of the vector entriesrespectively relate to the orthogonal spatial axes of the inertialsystem 13 of the camera 7. The angular orientation of the camera canalso be determined by evaluation of previous angular position values, bya compass, a Hall probe and/or a gravitation based inclination sensor.

The electronic components of the pocket computer 3 are constructed suchthat they can carry out the numerical integration of the measurementsdelivered by the acceleration sensors in order to determine the spatialposition and orientation of the camera. For this purpose, asix-dimensional speed vector, which contains three translational speedvalues and three rotational speed values, is generated from the sixorthogonal acceleration measurements of the acceleration vector by meansof the first integrator 11. Via the change of the acquired image dataand taking account of the acceleration measurements ascertained by theacceleration sensors coupled with the camera, the current speed of thecamera can also consequently be determined. Here, a speed-offset can betaken into account by a simple vector addition of a six-dimensionalspeed-offset vector and in this way be cancelled from the calculations.Input means can also be provided, through which the measurement of theacceleration sensors and/or the angular acceleration sensors is set tozero.

By means of the second integrator 12, a six-dimensional relativeposition vector is generated from the speed vector, which specifies thedeviation of a previous position of the camera to a subsequent cameraposition. This six-dimensional relative position vector contains threelocation differential coordinates (Δx, Δy, Δz), which specify thedeviation of the location coordinates (x, y, z) of the previous cameraposition to the following camera position, and three angulardifferential coordinates (Δφx, Δφy, Δφz), which specify the deviation ofthe angular position (φx, φy, φz) of the previous camera position to thesubsequent camera position. The integration may take place in discretetime increments or may also be carried out continuously, whereby thetime integration increments in the integration of the accelerationmeasurements should be smaller than the time period within which theacceleration of the camera changes substantially.

FIG. 6 shows a schematic representation of a movement trace with anumber of different successive positions of the vision aid systemaccording to the present invention. With the vision aid system accordingto the invention, a sequence of at least two images can be acquired tothe camera in respectively different positions of the camera or asequence of images in fixed time intervals at respectively differentposition of the camera can be recorded. From such a sequence of imagesacquired in relatively short time secession after one another withdifferent camera positions, a stereoscopic reproduction of the acquiredimages, i.e. a three-dimensional reproduction of the acquiredenvironment, can be calculated and visualized by means of the electronicmeans.

In addition, from the sequence of the acquired images from differentcamera positions, the spatial position and/or orientation of the camera7 is calculated by means of the electronic means. By use of locationdifferential coordinates and angular differential coordinates, therelative position of the camera 7 in reference to the camera position orthe position of the user of the vision aid system at the acquisition ofthe previous or successive image can be assigned to each acquired imageof the sequence. In doing so, older images of a sequence of acquiredimages can be weighted increasingly weakly in their influence on thecalculations or may have no further influence on the calculations. Inthis way, a three-dimensional reproduction of the acquired images andthe objects located therein can be calculated and visualized with thevision aid system according to the invention with only one camera 7based on a sequence of serially acquired images from different camerapositions.

FIG. 7 shows a schematic representation of the vision aid systemaccording to a further preferred embodiment of the present inventionhaving two cameras 7. The second camera is integrated in the spectaclesat a position spaced from the first camera. With the aid of both cameras7, two images can be simultaneously acquired and, by means thereof, astereoscopic image generated. With the acquired stereo images, athree-dimensional reproduction of the acquired image and the objectslocated therein can be calculated by means of the electronic means ofthe vision aid system and visualized to the user. The calculation of thethree-dimensional reproduction of the acquired image can be carried outby application of the batch compensation method.

FIG. 8 shows a schematic representation of the vision aid systemaccording to a further preferred embodiment of the present inventionwith sensor arrays. In this embodiment, multiple distance sensors, whichare arranged in a shape of a one-dimensional or multi-dimensional array14, are provided on the spectacles 2. The sensors may thereby serve todetermine the distance of objects and/or obstacles in the acquired imageand to make this available to the vision aid system as additionalinformation. The distance sensors may be constructed as ultrasoundsensors or rotating laser scanners in the visible or invisiblewavelength range. Other types of sensors or actuators can also beprovided in the spectacles 2, such as e.g. ultrasound sensors, infraredsensors, photo-sensors, lasers or combinations thereof.

FIG. 9 shows a schematic representation of a further preferredembodiment of the vision aid system according to the invention. In thispreferred embodiment, the vision aid system is equipped with a lightsource 15 in order to illuminate the image area to be acquired by thecamera with light 18 in the visible or invisible wavelength range, withinfrared light, or with infrared or visible laser light 18, while thecamera 7 of the vision aid system is sensitive to the correspondingwavelength range. By means of such a light source 15, the image area tobe acquired by the camera can be illuminated with a homogeneous orstructured stripe pattern. For projection of a structured stripe pattern16 onto the image area to be acquired by the camera 7, a laser projectorwith a suitable deflection unit may also be used, a grid projector or aprojection mask which is positioned in the beam path of a light source.Interference effects can also be generated with laser light andprojected onto the image area to be acquired by the camera.

FIG. 9 shows a schematic representation of a vision aid system, which isequipped with a grid projector. The grid projector exposes the imagearea to be acquired by the camera with a stripe or grid pattern, whichfalls on the objects in the image area and is reflected there-from.Distortions of the stripe or grid pattern thereby result, from whichconclusions on the position, size and situation of the objects in theacquired image can be drawn. With the aid of the electronic means of thevision aid system 1, a three-dimensional reproduction of the acquiredimage and the objects located therein can be generated and visualizedfrom the distortions, deflections, displacements or other changes of thestripe or grid pattern.

The vision aid system can record alternately an image of the image areato be acquired under illumination with a stripe pattern 16 and an imageof the image area to be acquired without stripe pattern 16, and combineor compare the image data of the acquired images with one another bymeans of the electronic means. The image area to be acquired may bescanned by means of a column- and line-by-line controlled laser beam 18in order to obtain additional image data and/or information about thedistance of objects in the acquired image. At the same time, the visionaid system 1 according to the invention may be equipped with aphoto-detector, which determines the spatial position of objects in theacquired image and/or their distance to the camera, in that thephoto-detector carries out a time duration measurement of the laserlight 18 reflected by a related picture element.

LIST OF THE REFERENCE CHARACTERS

1 vision aid system

2 spectacles

3 separate electronic unit or pocket computer

4 cable connected data conductor and/or power conductor between 2 and 3

5 eye of the user

6 implantable stimulation device

7 camera

8 wireless data conductor between 2 and 3

9 sensors for measurement of the translational or angular acceleration

10 sensor for measurement of the translational and angular acceleration

11 electronic means or electronic integrator I

12 electronic means or electronic integrator II

13 inertial system of the camera

14 array of sensors

15 light source or grid projector

16 projected grid or stripe pattern

17 distortion in the reflected grid or stripe pattern

18 light beams or laser-light beams

1. A vision aid system comprising: at least one first camera foracquisition of an image having a plurality of image data; electronicmeans for processing the acquired image data; and a stimulation deviceimplantable in the body at the vision organ for reproduction of theprocessed image through corresponding stimulation of the vision organ,wherein additional information regarding the spatial position and/orparticular attributes of objects in the acquired image is visualized inthe reproduction of the processed image, which information is associatedwith objects in the acquired image, wherein a change of the position [ΔxΔy Δz Δφ_(x) Δφ_(y) Δφ_(z) ] of the camera relative to a previousposition of the camera is determined by means of the electronic meansthrough double integration of acceleration measurements delivered by anumber of acceleration sensors, and wherein the time integrationincrement in the integration of the acceleration measurements is smallerthan the time span within which the acceleration of the camera changessignificantly.
 2. A vision aid system comprising: at least one firstcamera for acquisition of an image having a plurality of image data;electronic means for processing the acquired image data; and astimulation device implantable in the body at the vision organ forreproduction of the processed image through corresponding stimulation ofthe vision organ, wherein additional information regarding the spatialposition and/or particular attributes of objects in the acquired imageis visualized in the reproduction of the processed image, whichinformation is associated with objects in the acquired image, wherein achange of the position [Δx Δy Δz Δφ_(x) Δφ_(y) Δφ_(z) ] of the camerarelative to a previous position of the camera is determined by means ofthe electronic means through double integration of accelerationmeasurements delivered by a number of acceleration sensors, and wherein,by means of the electronic means, a relative position vector isdetermined from the acceleration measurements delivered by a number ofacceleration sensors, which vector contains location differentialcoordinates (Δx, Δy, Δz) that specify the deviation of the locationcoordinates (x, y, z) of a previous camera position to the subsequentcamera position, and contains three angular differential coordinates(Δφ_(x), Δφ_(y), Δφ_(z)) that specify the deviation of the angularposition (φ_(x), φ_(y), φ_(z)) of the previous camera orientation to thesubsequent camera orientation.
 3. A vision aid system comprising: atleast one first camera for acquisition of an image having a plurality ofimage data; electronic means for processing the acquired image data; anda stimulation device implantable in the body at the vision organ forreproduction of the processed image through corresponding stimulation ofthe vision organ, wherein additional information regarding the spatialposition and/or particular attributes of objects in the acquired imageis visualized in the reproduction of the processed image, whichinformation is associated with objects in the acquired image, andwherein a change of the position [Δx Δy Δz AΔφ_(x) Δφ_(y) Δφ_(z) ] ofthe camera relative to a previous position of the camera is determinedby means of the electronic means through double integration ofacceleration measurements delivered by a number of acceleration sensors,wherein, for integration of the acceleration measurements, a numericalintegration method, the Euler method, the Runge-Kutta method, theBulirsch-Stoer method, the Adams-Gear method, or combinations of thesemethods are applied.
 4. A vision aid system comprising: at least onefirst camera for acquisition of an image having a plurality of imagedata; electronic means for processing the acquired image data; and astimulation device implantable in the body at the vision organ forreproduction of the processed image through corresponding stimulation ofthe vision organ, wherein additional information regarding the spatialposition and/or particular attributes of objects in the acquired imageis visualized in the reproduction of the processed image, whichinformation is associated with objects in the acquired image, wherein asecond camera is provided, which together with the first camera acquiresstereo images, wherein, by means of the electronic means with theacquired stereo images, a three-dimensional reproduction of the acquiredimage and the objects located therein is calculated and visualized, andwherein the electronic means carry out the calculation of thethree-dimensional reproduction of the acquired image by use of the batchcompensation method.
 5. A vision aid system comprising: at least onefirst camera for acquisition of an image having a plurality of imagedata; electronic means for processing the acquired image data; and astimulation device implantable in the body at the vision organ forreproduction of the processed image through corresponding stimulation ofthe vision organ, wherein additional information regarding the spatialposition and/or particular attributes of objects in the acquired imageis visualized in the reproduction of the processed image, whichinformation is associated with objects in the acquired image, andwherein input means are provided, through which the measurement ofacceleration sensors and/or angular acceleration sensors can be set tozero.
 6. A vision aid system comprising: at least one first camera foracquisition of an image having a plurality of image data; electronicmeans for processing the acquired image data; and a stimulation deviceimplantable in the body at the vision organ for reproduction of theprocessed image through corresponding stimulation of the vision organ,wherein additional information regarding the spatial position and/orparticular attributes of objects in the acquired image is visualized inthe reproduction of the processed image, which information is associatedwith objects in the acquired image, and wherein an angular orientationof the camera is determined by evaluation of previous angular positionvalues, by a compass, a Hall-probe and/or a gravitation-basedinclination sensor.
 7. A vision aid system comprising: at least onefirst camera for acquisition of an image having a plurality of imagedata; electronic means for processing the acquired image data; and astimulation device implantable in the body at the vision organ forreproduction of the processed image through corresponding stimulation ofthe vision organ, wherein additional information regarding the spatialposition and/or particular attributes of objects in the acquired imageis visualized in the reproduction of the processed image, whichinformation is associated with objects in the acquired image, andwherein the vision aid system alternately acquires an image of the imagearea to be acquired under illumination with a stripe pattern and animage of the image area to be acquired without illumination, andcombines or compares the image data of the acquired images with oneanother by means of the electronic means.